Information - Theoretic Limits for Density Estimation
نویسندگان
چکیده
This paper is concerned with the information-theoretical limits of density estimation for Gaussian random variables with data drawn independently and with identical distributions. We apply Fano’s inequality to the space of densities and an arbitrary estimator. We derive necessary conditions on the sample size for reliable density recovery and for reliable density estimation. These conditions are true simultaneously for both finitely and infinitely dimensional density spaces.
منابع مشابه
Information Theoretic Clustering using Kernel Density Estimation
In recent years, information-theoretic clustering algorithms have been proposed which assign data points to clusters so as to maximize the mutual information between cluster labels and data [1, 2]. Using mutual information for clustering has several attractive properties: it is flexible enough to fit complex patterns in the data, and allows for a principled approach to clustering without assumi...
متن کاملOn the role of MMSE estimation in approaching the information-theoretic limits of linear Gaussian channels: Shannon meets Wiener
This paper explains why MMSE estimation arises in lattice-based strategies for approaching the capacity of linear Gaussian channels, and comments on its properties.
متن کاملEstimation of Information Theoretic Measures for Continuous Random Variables
We analyze the estimation of information theoretic measures of continuous random variables such as: differential entropy, mutual information or KullbackLeibler divergence. The objective of this paper is two-fold. First, we prove that the information theoretic measure estimates using the k-nearest-neighbor density estimation with fixed k converge almost surely, even though the k-nearest-neighbor...
متن کاملSome Equivalences between Kernel Methods and Information Theoretic Methods
In this paper, we discuss some equivalences between two recently introduced statistical learning schemes, namely Mercer kernel methods and information theoretic methods. We show that Parzen window-based estimators for some information theoretic cost functions are also cost functions in a corresponding Mercer kernel space. The Mercer kernel is directly related to the Parzen window. Furthermore, ...
متن کاملProbability Density Estimation using Isocontours and Isosurfaces: Application to Information Theoretic Image Registration
We present a new, geometric approach for determining the probability density of the intensity values in an image. We drop the notion of an image as a set of discrete pixels, and assume a piecewise-continuous representation. The probability density can then be regarded as being proportional to the area between two nearby isocontours of the image surface. Our paper extends this idea to joint dens...
متن کامل